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ABSTRACT 

This qualitative study was designed to identify a framework for the quality of OBL in adult education (AE), 
which are of interest to the needs of students. Following a review of the literature, we opted for the theoretical 
framework as proposed by Ossiannilsson and Landgren (2012). This framework suggests success factors for 
OBL that are of interest to the needs of students. Qualitative data was collected through group interviews (n=12 
groups) with stakeholders involved in AE. Professionals from five institutions, at the policy level (n=17) or 
programme level (n=20) were interviewed. Findings were discussed and agreed upon by the researchers to 
validate a quality framework for OBL in AE. At the level of the success factors, it is useful to underpin the 
adoption of OBL, to formulate a mission statement. The indicators can help set goals, identify resources and 
strategies and measure whether the provision aligns with adult students’ needs. All success factors and indicators 
are linked to quality areas and dimensions most existing quality frames are built on. At this level the framework 
is useful to mainstream the quality of OBL into traditional frameworks. 

Keywords: Quality, Success Factors, Blended Learning, E-learning, Online Learning, Quality enhancement. 

INTRODUCTION 

Online and blended learning (OBL) is highly valued in education but at the same time challenges educational 
institutions. It is valued e.g. to enhance accessibility and flexibility in education (Graham & Robison, 2007; 
Shea, 2007), to reduce the costs of instruction (Shea, 2007) and even to transform traditional approaches to 
instruction and teaching(Garrison & Kanuka, 2004; Graham & Robison, 2007). OBL is appealing for institutions 
in order to tailor the educational provision to the needs of (adult) students in terms of pedagogy and to support 
them to balance education with other responsibilities. OBL is equally highly challenging for institutions because 
quality assurance and improvement (QA&I) requires a systemic approach and the involvement of many, 
including students (Deepwell, 2007; Jara & Mellar, 2009). Yet it is not easy in education to consult students, and 
even more difficult in the case of OBL as a result of the limited presence of students (Bloxham, 2010; Jara & 
Mellar, 2009). 

Knowledge of what defines quality of OBL from the student perspective is therefore beneficial but was lacking 
until recently. This knowledge can support institutions to underpin the adoption of OBL and set goals, identify 
resources and strategies and measure whether the provision aligns with (adult) students’ needs. 

We firstly present commonalities in existing quality frameworks for OBL from the providers’ perspective 
reported in literature. Then we will discuss aconceptual framework for quality of OBL to meet students’ 
needs. Finally,we will address both empirical studies and studies on existing quality models that define quality 
of OBL for (adult) students in the context of HE in relation to the conceptual framework and the commonalities 
in existing quality frameworks. 

LITERATURE REVIEW ON QUALITY FRAMEWORKS FOR OBL 

With the rise of OBL in HE, the issue of quality assurance and improvement (QA&I) was raised)Jara & Mellar, 
2009). Institutions, governments and QA agencies needed to know how to assess the quality of OBL (Jara & 
Mellar, 2009). In order to meet this need, several quality frameworks were developed for OBL-education in 
higher education (HE) worldwide (Ossiannilsson, Williams, Camilleri & Brown, 2015). These frameworks differ 
by scope, structure, type of institution they are aimed at and intended way of use (Inglis, 2005). Nonetheless the 
differences between quality frames, scholars noted that they have similar "constituents ’(Frydenberg, 2002; Jung, 
2011; Phipps & Merisotis, 2000). According to Ossiannilsson and Landgren (2012) these can be clustered in 
three quality areas (management, services and products) and six quality dimensions: management, student 
support, support to teachers and staff, curriculum design, course design and delivery. 
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Quality of OBL from the providers’ perspective in Higher Education 

The concept of quality dimensions originates from the total quality movement (TQM) in industry during the 
previous century (Srikanthan & Dalrymple, 2002). Srikanthan and Dalrymple (2002) advocate for holistic 
quality models that address service and pedagogical aspects. Quality frameworks are traditionally presented from 
an institutional perspective by the quality areas and quality dimensions (Frydenberg, 2002; Jung, 2011; Phipps & 
Merisotis, 2000). This is important because the deployment and QA&I process (of OBL) demands a holistic, 
often also called systemic or process approach. The consensus in the scientific community for this statement 
(Hansson, 2008; Ossiannilsson et al., 2015) is clearly expressed by Hansson(200Sa): “When implementing e- 
learning, it is important to adopt a holistic approach. ... aspects ... are part of a puzzle in which all the pieces 
have to fit together. When one part of the puzzle changes, e.g. technology’, student behaviour, knowledge needs, 
society, finances or staff requirements, all other parts needs to be re-aligned accordingly” (Hansson, 2008a, p. 
56). 

TQM frameworks cover management processes, but miss the focus on the student learning experience 
(Srikanthan & Dalrymple, 2002). Dumont and Sangra (2006) came to the same conclusion when assessing 
different quality frameworks in European HE. Scholars report that quality frameworks in HE are often conceived 
from the perspective of the provider (i.e. institutions, government) (Frydenberg, 2002; Jung, 2011). Ehlers and 
Pawloski (2006) state that the student perspective of quality does not necessarily coincide with other 
stakeholders’ views.In view of recent literature which emphasises that educational quality is the result of a 
negotiation process between all participating parties in education (Ehlers, 2009a, 2009b) the student perspective 
is important. Moskal et al. (Moskal, Dziuban, & Hartman, 2013) refer to alignment as ‘institutional alignment’, 
which implies that institutional (meso level), faculty (micro level) and student goals are aligned. Institutional 
alignment requires a culture focused on quality in which key stakeholders actively participate in dialog (Ehlers, 
2009a, 2009b; Moskal et al., 2013). However, the limited opportunities to interact with students in OBL puts 
pressure on their involvement (Jara & Mellar, 2009). It is thus important that quality frameworks comply with 
the needs of students. However, the question to what contributes to quality of OBL from a student’s point of 
view remained unanswered until recently. 

Quality of OBL from the students perspective in Higher Education 
Conceptual framework of Ossianilsson & Landgren (2012) 

The work of Ossiannilsson and Landgren (2012) focuses on the quality frameworks for OBL in HE. These 
authors compared the output of international benchmarking projects, the e-learning quality model (ELQ) 
outlined by the Swedish National Agency for Higher Education (NAHE) (Hansson, 2008) and analysed literature 
from that comparison. A conceptual framework with a range of critical success factors: accessibility, flexibility, 
interactivity, transparency, participation, personalisation and productivity for quality in e-learning to meet 
students’ needs emerged (Ossiannilsson & Landgren, 2012). The latter three success factors are related to 
pedagogy (McLoughlin & Lee, 2008). Success is described by Ossiannilson and Landgren (2012) as: ‘to be 
successful in e-learning from an academic and educational point of view but also with regard to their personal 
and social life ’(Ossiannilsson & Landgren, 2012, p. 49). The authors (Ossiannilsson & Landgren, 2012) suggest 
that the success factors should be embedded in all quality areas and quality dimensions: managerial levels 
(strategic planning and development), services (staff support and student support) and products (curriculum and 
course design, course delivery). 

This conceptual framework presents a view to what ‘ constituents’ add to quality of OBL when looked at from 
the students’ perspective. While the authors describe the success factors, definitions of the concepts are not 
given. How the quality dimensions are connected to the success factors is neither made explicit. The self- 
assessment tool e-xcellence (Kear et al., 2016; Ubachs et al., 2007; Williams, Kear & Rosewell, 2012), 
developed by the European Association of Distance Teaching Universities (EADTU), is built upon this 
conceptual framework and contains benchmarks. Yet, the assessment tool is presented from the traditional 
institutional perspective i.e. quality areas and quality dimensions instead of the students’ perspective. 

Empirical studies on quality dimensions from the (adult) student perspective in Higher Education 

The scarce studies in search for quality dimensions of OBL from the (adult) students’ perspective indicate that 
the framework of Ossiannilsson and Landgren these (2012) can be fine-tuned for application in the context of 
adult education. While several of the quality dimensions reported in these studies coincide with the success 
factors reported by Ossiannilsson and Landgren (2012) also differences appear. 
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In an empirical study Ehlers (2004) interviewed experienced students in OBL in the European context of HE. 
From these interviews Ehlers (2004) constructed a questionnaire answers of students were analysed by principal 
component analysis and cluster analysis, seven quality fields (key factors): tutor support, cooperation, 
technology, costs-expectations-value, information transparency, course structure and didactics, and thirty quality 
dimensions for OBL were identified. Several of these quality fields, with their underlying quality dimensions 
seem to coincide with the success factors reported by Ossiannilsson and Landgren (2012) (see tTable 1). Ehlers 
(Ehlers, 2004) identified different student preference profiles for the quality of OBL. Although the perception of 
quality can differ between students it can be defined by a limited number of quality fields or ‘ constituents 

Table 1: Comparison of success factors reported by Ossiannilsson and Landgren (2012) with reported quality 
fields and quality dimensions by Ehlers (2004) and quality dimensions by Jung (2011). 

Ossiannilsson and Landgren (2012) 

Ehlers (2004) 

Jung (2011) 

Reported success factors 

Reported 

Reported quality 


quality fields or quality dimensions 

dimensions 

- 

- 

Institutional credibility 

A. Flexibility 

- 

- 

B. Transparency 

Information transparency (QF 5) 

Information and publicity 

C. Accessibility 

- 

- 

D. Personalisation 

Student vs. Content centeredness (D3) 

- 

E. Interactivity 

Collaboration (QF 2) 

Interaction centeredness (D1) 

Interaction 

F. Productivity 

- 

- 

G. Participation 

- 

- 

Quality areas/Quality dimensions 

Reported 
quality’ fields 


management 

• Strategic planning and 

- 

Institutional QA 

development 


mechanism 

Services 

• Student support 

Tutor support (QF 1) 

Student support 

• Support to teachers and staff 


Staff support 

Products 

- 

- 

• Programme design 

- 

- 

• Course design 

Course structure (QF 6) 

Didactics (QF 7) 


- 

- 

Learning tasks 

• Delivery 

Technology (QF 3) 

- 

- 

Costs - expectations - value (QF 4) 
Moderation of learning processes (D2) 

- 


Based on literature Jung (2011) developed a questionnaire with seven quality dimensions: institutional support, 
course development, course structure, teaching and learning, student support, faculty support, and evaluation and 
assessment. Responses of students familiar with OBL were analysed by exploratory and confirmatory factor 
analysis to determine the number of common factors and to examine the structure of those factors and 
intercorrelations among them. Jung (2011) empirically identified the following quality dimensions important to 
Korean adult students in HE: information and publicity, student support, staff support, institutional quality 
assurance mechanisms, institutional credibility, learning tasks and interaction. Several of these quality 
dimensions also coincide with the success factors reported by Ossiannilsson and Landgren (2012) (seeTable 1). 

Next to the similarities presented above, differences also emerged. No reference towards attributes 
foraccessibility is found in Jungs’ study (2011) and Ehlers (2004) indicates that students acknowledge the 
importance of technology for quality of OBL only when it is lacking. Productivity and participation were also 
not mentioned by Jung (2011) and Ehlers (2004). 

One might expect that flexibility is a prerequisite for the quality of OBL, crucial for adult students to keep 
professional obligations balanced with educational engagement. However, indicators related to flexibility e.g. 
‘flexibility of the learning pace’ were excluded from the final indicator list in Jungs’ study (2011). Little 
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reference towards attributes for flexibility is found in Ehlers (2004) study. Inglis (2008) indicates that the 
validation processes of quality frameworks take place against (geographical) contexts which have an impact on 
the result(ing) current practices, e.g. the way OBL is conceived in a region, can have an impact on what is 
reported as important. This might be the reason why the indicator ‘flexibility of the learning pace’ was excluded 
in Jungs’ study (2011) and not reported by Ehlers (2004) as both studies were conducted in the context of HE. A 
context in which the flexibility of the learning pace for students is likely limited. Contextualization of the 
findings to the context of adult education seems thus necessary. 

Another difference reported by Jung (2011) relates to the quality dimension ‘course design’. It appeared that 
content and structure of courses was not important for adult students per se. Jung (2011) concluded that the 
design of learning tasks might be crucial for adult students instead. It should be noted that attributes of learning 
tasks (Jung, 2011) are similar to the concept of personalization: Problem-based learning tasks, individualized 
learning tasks and collaborative learning tasks.Neither assessment and evaluation nor technological support 
appeared to be critical for adult students (Jung, 2011). 

Jung (2011) reports that ‘institutional quality assurance (QA) mechanism’ and ‘institutional credibility’ were 
important for adult students. These were not reported by Ehlers (2004).Respondents valued that the institution 
was accredited at the national level and had installed a QA policy with clear policies and guidelines (Jung, 
2011). It is not clear how ‘institutional credibility’ is to be seen. It can be either interpreted as a quality 
dimension i.e. ‘management’ in Ossiannilsson and Landgren’s (2012) conceptual model or a new success factor 
i.e. ‘credibility’. Credibility, according to Jung (Jung, 2011), stands for external accreditation, international 
recognition and strong leadership. 

Studies on existing quality models, their quality aspects and quality indicators that define quality of OBL 
for adult students in the context of HE. 

Findings in Ehlers (2004) and Jungs’ (2011)study support the claim of Ossiannilsson and Landgren (2012) that 
several success factors for quality of OBL in HE exist. Studies in HE about existing quality models/systems or 
quality aspects/indicators that define quality of OBL for AE provide additional support (Table 2). 


Table 2: Overview of articles (author and title) and comparison with success factors reported by Ossiannilsson 

and Landgren (2012) and Jung (2011)*. 


nr. 

Author 

Title 

Success factors 




C* 

F 

T 

A 

I 

Pe 

Pr 

Pa 

i. 

Korres, Karalis, 
Leftheriotou, & 
Barriocanal (2009) 

Integrating Adults' Characteristics and the 
Requirements for Their Effective Learning in an 
e-Learning Environment 

X 




X 

X 

X 


2. 

Dzakiria (2012) 

Illuminating the importance of learning 
interaction to open distance learning (ODL) 
success: a qualitative perspectives of adult 
learners in Perlis, Malaysia 





X 




3. 

Zhang & Cheng 
(2012) 

Quality assurance in e-learning PDPP evaluation 
model and its application 

X 








4. 

Volungeviciene, 
Tereseviciene, & Tait, 
(2014) 

Framework of quality assurance of TEL 
integration into an educational organization 

X 



X 



X 


5. 

Stodel, Thompson, & 
MacDonald (2006) 

Learners' perspectives on what is missing from 
online learning: interpretations through the 
community of inquiry framework 

X 



X 

X 

X 



6. 

MacDonald & 
Thompson (2005) 

Structure, content, delivery, service and 
outcomes: Quality e-learning in higher education 




X 

X 

X 



7. 

Harroff, P.A. (2002) 

Dimensions of quality for web-based adult 
education 

X 


X 

X 






Credibility, Volungevience et al. (2014)provide an argument to see ‘credibility' as a new success factor related 
to the quality dimension ‘management’ because they plead for a clear vision on the reason for implementating 
OBL. This is in line with Zhang and Cheng (2012) who advocate for clear OBL implementation processes and 
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guidelines. Strong leadership is apparent when teachers who experiment and implement OBL are credited 
(Harroff, 2002). Besides strong leadership, installation of adequate resources is considered necessary by several 
authors, ranging from staff support through clear roles (Korres, Karalis, Leftheriotou, & Barriocanal, 2009; 
Volungeviciene et al., 2014), adequate technical training (Harroff, 2002) and training in applying new pedagogy 
(Korres et al., 2009; Stodel, Thompson, & MacDonald, 2006). Finally, credibility refers to an internal QA 
system with a focus on written QA guidelines for OBL (Jung, 2011), specific quality measures (Jung, 2011; 
Volungeviciene et al., 2014; Zhang & Cheng, 2012) and consultation with different stakeholders such as 
students (Harroff, 2002) and staff (Jung, 2011). All these quality attributes are important to install a sustainable 
OBL programme, and can be considered management responsibilities, no different from a standard education 
programme, but with a specific focus for OBL. It is clear that this success factor is important from a provider’s 
perspective and is indirectly relevant for students. However Jung’s (2011) study indicates that adult students also 
consider these attributes important. 

Transparency is important for adult students because it gives them a clear idea of what to expect and to 
consider if the programme can be combined with their professional and personal occupations. It is crucial to 
provide correct information, before and during the programme, about: admission requirements (Harroff, 2002), 
costs (Harroff, 2002; Jung, 2011), programme length, expectations of technical knowledge and information on 
course requirements (Harroff, 2002). 

Accessibility can be looked at from different perspectives. Products such as courses should be easily accessible 
and easy to use (MacDonald & Thompson, 2005; Volungeviciene et al., 2014). This is equally true for services 
such as learning support (Harroff, 2002; MacDonald & Thompson, 2005) that are provided. Stodel et al. (2006) 
focus on accessibility of the design on a deeper, pedagogical, level. The use of technologymust be in line with 
the chosen pedagogy by using appropriate tools e.g. tools for synchronous and asynchronous communication. 

Personalisation seems also present in the literature on OBL in AE. It ranges from content (Korres et al., 2009) 
or design (Stodel et al., 2006) that meets expectations and interests of the students, over providing authentic and 
personally meaningful problems (Jung, 2011), to even rapid redesign to adapt to the students’ needs (MacDonald 
& Thompson, 2005). Not only does the design refer to personalisation, also the students’ support is personalised 
or need based (Jung, 2011). 

The instructional design needs to provoke activation or interactivity of the students with the materials by 
providing authentic materials (MacDonald & Thompson, 2005) or to initiate interaction between students by 
collaborative tasks (Korres et al., 2009) and meaningful dialogue (MacDonald & Thompson, 2005). The student- 
student interaction is also intended as technical support (Stodel et al., 2006) or emotional support (MacDonald & 
Thompson, 2005). Finally literature refers to student-teacher interaction with a focus on the quality of the 
feedback from the tutors (MacDonald & Thompson, 2005). 

Productivity is mentioned in relation to an indicator i.e. problem based learning (Jung, 2011) or higher-order 
thinking (analysis, synthesis, evaluation) (Korres et al., 2009) and complex tasks integrated with assessments 
(Volungeviciene et al., 2014). 

The way McLoughlin and Lee (McLoughlin & Lee, 2008) describe participation (communication, 
collaboration, connectivity and community) is similar to the way interactivity is described by Ossiannilsson and 
Landgren (Ossiannilsson & Landgren, 2012) (interactivity with content, peers and teachers). It appears thus that 
‘participation’ and ‘interactivity’ are closely related. However, both success factors do not seem interchangeable. 
It can be argued that participation is key to turning all factors into success factors for OBL given that the right 
decisions are made, either by enabling participation (flexibility, accessibility, transparency) in education or by 
inviting students to participate actively in the learning process (interactivity, personalisation, productivity). This 
is in line with the concepts of enabling blend and transforming blend reported by Graham (Bonk & Graham, 
2012; Graham, 2005; Graham & Robison, 2007). 

In summary, it appears that frameworks for quality of OBL are often conceived and presented from the 
perspective of the provider (i.e. institutions, government, QA agencies), lacking the student perspective 
(Frydenberg, 2002; Jung, 2011). Yet in education the concept of quality is a client-oriented i.e. a student oriented 
concept defined by dialogue between students and providers (Elders, 2007). Because it is not easy ineducation to 
consult students in the quality dialogue, and even more difficult in the case of OBL as a result of the limited 
presence of students (Bloxham, 2010; Jara & Mellar, 2009) this is an issue. 
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Despite all differences between quality frameworks for OBL (Inglis, 2005) it appears they have similar 
‘constituents ’ (Frydenberg, 2002; Jung, 2011; Phipps & Merisotis, 2000), which can be clustered in three quality 
areas and six quality dimensions (Ossiannilsson & Landgren, 2012). Scientific knowledge about the 
‘ constituents’ of quality of OBL from the students perspective and how these are related to quality areas and 
quality dimensions present in most current quality frameworks is lacking. This knowledge is beneficial because 
it can allow institutions to underpin the adoption of OBL, set goals, identify resources and strategies and 
ultimately measure whether their provision is tailored to the needs of (adult) students. 

Until recently it was not clear how quality of OBL from the students’ perspective could be defined. The 
conceptual framework by Ossiannilsson and Landgren (2012) provides an answer to this question. Although 
literature supports it, the framework remains conceptual. It appears that until now HE institutions have been the 
central focus in research regarding the use of quality frameworks in education (Contreras, Torres, Palominos & 
Lippi, 2015). It is not clear if the findings from studies (with adult) students in HE are transferable to students in 
other educational contexts. 

This study is designed to provide scientific knowledge about the ‘ constituents' by which students measure 
quality of OBL in adult education and how these are related to quality dimensions present in most quality 
frames. The work of Ossiannilsson and Landgren (Ossiannilsson & Landgren, 2012) is chosen as a theoretical 
framework. 

The research questions in this study are: 

1. Which success factors are essential for the success of OBL in adult education as perceived by adult 
education stakeholders (RQ1)? 

2. Which quality areas and dimensions are essential for the success of OBL in adult education as perceived by 
adult education stakeholders (RQ2)? 

3. Which quality framework can be validated for OBL in AE and which indicators for quality can be identified 
(RQ3)? 

METHODOLOGY 

According to Inglis (2008), quality frameworks can be validated by either reference to appropriate research 
literature or against the knowledge of experts in the field or through a combination of both. While it is 
appropriate to draw on literature it may not be sufficient especially in new contexts (Inglis, 2008). Stakeholders 
can be assembled to elicit their expert knowledge, which is tacit as well as explicit (Inglis, 2008). Therefore, 
qualitative data were drawn from group interviews (n=12 groups) in five institutions in addition to literature to 
validate the framework. Professionals were interviewed at the policy level (n=17) and programme level (n=20). 

The principles of thematic analysis is chosen as a method. This method allows to combine deductive matrix 
analysis with the principles of grounded theory (Corbin & Strauss, 1990). Grounded theory is a method which is 
based on inductive analysis from the data focused on creating conceptual frameworks (Charmaz, 2006). 

Research context 

This study was conducted in Flanders (the Dutch speaking part of Belgium). The Flemish government, operating 
at the macro level, promotes OBL in AE to satisfy the demand for flexible education for adults (Decree 
15/06/2007). The Flemish Inspectorate is responsible for the external evaluation of institutions in this context. 
The Context, Input, Processes, and output model (CIPO-model) (Scheerens, 1990, 2006) is a generic quality 
model that is used by the Flemish Inspectorate to perform quality audits in different educational contexts 
amongst which are adult education institutions. Currently the quality of OBL in AE institutions is assessed 
separately by the Inspectorate based on a minimal set of criteria by decree (Decree 15/06/2007). Another 
governmental body, ‘verification’, exerts control on attendance of adult students to the educational provision on 
which institutional funding is based. 

While external quality control of OBL is not part of the regular procedure used by the Inspectorate their reports 
reveal information about the centers with respect to their internal QA&I. Inspection reports (N—4) not older than 
five years of the centers involved in this study were scanned. Analysis of these indicates that: '... Digital 

learning in specific courses . currently internal quality is implemented at institutional level, the institution is 

still looking ■■■for indicators to measure the educational process. Align the (internal) quality’ assurance with 
monitoring the quality of the core process 

Institutions, operating at the meso level, can apply for a financial incentive if they choose to adopt OBL in their 
educational provision. Beyond adoption of OBL, institutions of AE are challenged to incorporate quality of OBL 
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into the institutional quality procedure. They have been compelled to examine and systematically monitor the 
quality of their provision since 2009(Decree08/05/2009). Institutions offer programmes for a diverse audience, 
from participants in basic education (primary and middle school education for adults), (vocational) second 
chance education, to courses at Level 5 of the European Qualification Framework and teacher training.For 
QA&I of OBL institutions either turn to what is available in HE or adopt an ad hoc approach. They need to 
know how to mainstream the quality of OBL into their implemented QA&I approach. 

Procedure 

To explore the current approaches and experiences with QA&I, qualitative data were drawn from semi- 
structured interviews in five institutions for AE. In each institution an interview was conducted with policy 
makers and QA coordinators (n=17) followed by an interview with professionals at the programme level (n—20). 
Respondents were interviewed about current approaches and experiences with QA&l in general and OBL 
specifically. Although the interviews focused on QA&I, the topic of the quality of OBL was omnipresent, 
interviewees talked about the quality of OBL in adult education. 

The interview guideline was structured by the PDCA-cycle (Plan, Do, Check, Act) (Deming, 1950) and 
addressed topics regarding QA&I of OBL in the institution: institutional policy, implemented framework 
(including quality domains and indicators), implementation of QA&I in the institution and involvement of 
different stakeholders, effect and impact of quality assurance and plans for improvement. 

Participants 

Principles of theoretical sampling (Corbin & Strauss, 1990) were used to select centers and respondents. To be 
selected, centers had to: (a) currently provide (or have provided in the past) part of their provision through OBL; 
(b) have experience with OBL exceeding more than one year. Sample characteristics are displayed in Table 3. 
Centers were contacted and informed about the purpose of the study and about criteria for the inclusion of 
respondents. The researchers aimed for programmes at secondary education level and at Level 5 of the European 
Qualification Framework. 


Table 3: sample characteristics of centers. 



Case a 

Case b 

Case c 

Case d 

Case e 

Total 

Number of respondents at policy level 

N=3 

N=3 

N=3 

N=5 

N=3 

N=17 

Number of respondents at programme level 

Interviews 

N=3* 

N=j** 

N=7** 

N=2** 

n =3** 

N=20 





N=2** 







N=2** 



* Level 5 of the European Qualification Framework, ** Secondary education 


Criteria for inclusion of respondents at the programme level were: (a) to have at least one year of experience 
with OBL; (b) all teachers participating in an interview had to teach in the same programme. For details of their 
experience in education and with OBL see Table 4. The selection of programmes and respondents was trusted to 
the institutions. One institution pointed out that in total three programmes were experienced with OBL. All were 
included to increase data saturation. 

The interviews (N=12) were conducted over a period of three months. During each interview two researchers 
were present. One acted as the moderator and one as the observer, which allowed the interviewer to focus on the 
discussion. The observer took notes and assured all topics were covered. The first author was present at all the 
interviews. Interviews were transcribed in full and are the focus of this analysis. 
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Table 4: Experience of respondents in years. 


Experience: 

<5y 

>* 

o 

’-H 

1 

>> 

so 

>> 

o 

<N 

1 

>> 

VJ 

1 

o 

>30y 

Institutional level 

In Education 

l 

i 

8 

4 

3 

In Current position 

4 

5 

7 

- 

- 

With OBL 

8 

8 

1 

- 

- 

Programme level 

In Education 

3 

2 

8 

3 

4 

In Current position 

3 

1 

14 

1 

1 

With OBL 

13 

5 

2 

- 

- 


Data analysis 

Interviews were coded and analysed by the first researcher, according to the coding scheme as proposed by 
Corbin and Strauss (1990): open, axial and selective coding. 

First all parts in the interviews in which respondents expressed anything that from their perspective was 
important for either OBL or QA&I were free coded (open coding). Only explicitly mentioned success factors 
were coded according to pre-deftned codes, but not limited to: flexibility, accessibility, transparency, 
interactivity, personalisation, productivity, participation (Ossiannilsson & Landgren, 2012). 

During a second phase, firstly, open codes were thematically clustered based on the quality areas and quality 
dimensions, but not limited to, the work of Ossiannilsson and Landgren (Ossiannilsson & Landgren, 2012): 
management (strategic planning and development), products (design - (curriculum/programme, course, learning 
activities and assessment) and delivery) and services (teacher and staff support, student support). Distribution 
(over cases and interviews) and frequency of coded statements were used as the criteria to identify themes.For 
inclusion, it was decided that themes were to be coded in at least four cases or seven interviews because of 
scientific consensus in literature about their existence. Then, axial and selective coding were performed to 
establish relationships between them and quality areas and dimensions. Thematically clustered codes were re¬ 
coded in terms of the success factors. Descriptions of these concepts that are mentioned in the e-xcellence 
manual (Kear et al., 2016; Ubachs et ah, 2007; Williams, Kear, & Rosewell, 2012)which is built upon the 
conceptual framework for quality in e-leaming developed by Ossiannilsson and Landgren (Ossiannilsson & 
Landgren, 2012) and the three P’s of pedagogy (McLoughlin & Lee, 2008) were used to guide the analysis. 

During a final coding round of the interviews, remaining codes were re-examined and coded in terms of success 
factors, after which seventy-four codes remained distributed over all cases and interviews. 

The results from the interviews and literature e.g. ‘Credibility’, ‘external QA’ and ‘internal QA’ (Jung, 
2011)were combined to validate the conceptual framework with success factors and determine indicators for 
OBL in AE. To enhance credibility results from the interviews were challenged and discussed with another 
researcher who was not involved in the analysis of the interviews. The processes from the CIPO-model were 
used to structure this iterative process. Concordance between researchers was reached by agreement about 
success factors, indicators and their links to quality areas and dimensions. The integration resulted in an adapted 
framework. 

RESULTS 

Findings from the interviews are presented in relation to each of the research questions. 

RQ 1 - Which success factors are essential for the success of OBL in adult education as perceived by adult 
education stakeholders? 

Except ‘productivity’ and ‘credibility’ all success factors were explicitly mentioned and coded during the first 
coding phase, but not all success factors were distributed (mentioned) equally over cases or interviews (see 
Table 5). While ‘flexibility’ and ‘personalisation’ were explicitly mentioned in all cases (resp. in ten and eight 
interviews), ‘interactivity’ and ‘participation’ were mentioned respectively in three cases (four interviews) and in 
three cases, ‘accessibility’ and ‘transparency’ were mentioned only once. 
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Table 5: coding by success factors by coding round. 


Success factors 

1st coding 
round 


2nd coding 
round 


Final coding 
round 



Distribution 

over cases, 
interviews 

Frequency 

Distribution 

over cases, 
interviews 

Frequency 

Distribution 

over cases, 
interviews 

Frequency 

Flexibility 

5,10 

74 

5,11 

110 

5,11 

121 

Accessibility 

u 

1 

3,7 

27 

3,7 

29 

Transparency 

1,1 

1 

5,11 

90 

5,11 

96 

Interactivity 

3,4 

14 

5,8 

33 

5,8 

33 

Participation 

3,3 

5 

4,4 

9 

4,4 

13 

Productivity 

0,0 

0 

4,4 

6 

4,4 

6 

Personalisation 

5,8 

22 

5,9 

29 

5,9 

29 

Integration 

- 

- 

4,6 

37 

4,6 

37 

Credibility 

- 

- 

- 

- 

- 

- 


During the second coding phase, remaining free codes were thematically clustered based on, but not limited to, 
the quality areas and quality dimensions reported by Ossiannilsson and Landgren (2012) and Jung (2011): 
Management (strategic planning and development), products (design - curriculum/course/learning activities and 
assessment; delivery) and services (teacher and staff support, student support), ‘external QA’ and ‘internal QA’. 
The thematically clustered free codes were then recoded in terms of the success factors. After this, distribution 
and frequency increased for all success factors except credibility. All were mentioned in at least three cases and 
at least four interviews. 

After this coding phase, remaining free codes and thematically clustered based were re-examined and coded in 
terms of success factors to establish relationships between the themes and the success factors. After this, third 
and final coding round, distribution and frequency increased for all success factors exept ‘credibility’. After this 
coding round seventy-four codes remained. 

RQ 2 - Which quality areas and dimensions are essential for the success of OBL in adult education as 
perceived by adult education stakeholders? 

Several themes emerged that could be categorised within Ossiannilsson and Landgrens’ (Ossiannilsson & 
Landgren, 2012)and Jungs’ (201 l)quality areas and quality dimensions; see Table 6. Two themes did not fit with 
the pre-defined quality areas and quality dimensions (n=4). These were labelled: ‘integration’ and ‘evolution’. 


Table 6: thematical clustering of elementary codes from interviews. 



Exter 

nal 

QA 

Managem 

ent* 

Teacher and staff 
support** 

Student 

support** 

Design 

*** 

Delivery 

•k'k'k 

Evoluti 

on 

Inter 

nal 

QA 

Distribut 









ion over 
articles 

4,4 

5,12 

5,11 

5,12 

5,11 

4,7 

4,7 

5,12 

Frequen 

JZ 

18 

103 

132 

198 

163 

44 

9 

68 


Ossiannilsson and Landgren (2012): Management* (strategic planning and development), products** (design - 
curriculum/course and assessment; delivery) and services *** (teacher and staff support, student support). 


By recoding the remaining free codes that were thematically clustered in terms of success factors, relationships 
between themes and success factors were established (Table 7). The ‘design’ of the provision (course, 
programme, learning activities and assessment) could be linked to ‘flexibility’, in total thirty-six statements, 
distributed over all cases and ten interviews. Both ‘student support’ (2,3-6) and ‘(online) delivery’ (3,6 -20) 
could be linked to ‘accessibility’. ‘Interactivity’ (nineteen statements distributed over all cases and seven 
interviews) appeared to be a success factor for the ‘design’ of the provision (course, programme, learning 
activities and assessment). ‘Design’ could also be linked with ‘personalisation’, (2,3-7) while only six statements 


Copyright © The Turkish Online Journal of Educational Technology 


170 





















TOJET: The Turkish Online Journal of Educational Technology - July 2017, volume 16 issue 3 



from the ‘design’ of the provision (course, programme, learning activities and assessment) could be linked to 
‘productivity’ (4,4). Both ‘student support’ (1,1-2) and ‘design’ (1,1-5) could be linked to ‘participation’. All 
quality dimensions, with the exception of ‘support for teachers and staff, are linked to ‘transparency’ 
(‘management’ 3, 3 - 7; ‘student support’ 5,10 - 49; ‘design’ 5,6 - 14 and ‘delivery’ 2,3 - 19). Within ‘design’ 
(course, programme, learning activities and assessment) thirty-seven statements (4,6) which were labeled as 
‘integration’ emerged which appeared to be important, but could not be linked to a success factor reported by 
Ossiannilsson and Landgren (2012). Management and the themes: ‘external QA, ‘evolution’, ’integration’ and 
‘internal QA’ could not be linked to any of the success factors. 


Table 7: connections between success factors and emerging themes. 


Success factors 

Management 

(5,12-103) 

Teacher and 
staff support 

(5,11-132) 

Student 

support 

(5,12-198) 

Design Delivery 

(5,11 - 163) (4,7-44) 


Distribution over cases, interviews - frequency 

Flexibility 
(5,11 - HO) 

- 

- 

- 

5,10-36 

Accessibility 

(3,7-27) 

- 

- 

2,3-6 

3,6-20 

Transparency 

(5,11-90) 

3,3-7 

- 

5,10-49 

5,6-14 2,3-19 

Interactivity 

(5,8-33) 

- 

- 

- 

5,7-19 

Participation 

(4,4-9) 

- 

- 

U-2 

1,1-5 

Productivity 

(4,4-6) 

Personalization 

(5,9-29) 

- 

- 


4,4-6 

2,3-7 

Credibility 

- 

- 

- 

- 

Integration 

(4,6-37) 

- 

- 

- 

4,6-37 


RQ3 - Which quality framework can be validated for OBL in AE and which indicators for quality can be 
identified? 

Flexibility and transparency are mentioned in all cases and interviews. The use of technology helps students to 
combine education with professional and private obligations. The way OBL is designed i.e. flexible deadlines 
for learning activities: ‘ ...if they give me a reason, it's good I'll mention: "has a reason. " or "will catch up later ', 
and the programme: ‘... there are not enough days in a week to actually do it all (ed. the programme,) in one 
year. Plus also the people who work, ... full time...', ‘we think if we are targeting students for distance learning 
in programme X, we’ll start with a percentage of OBL in the range from 25% to 35%’. ... ‘students ask for 
distance learning occasionally to not come some evenings’ ... ’yes’ ... ‘but they surely still want to come to 
class’. Transparency is seen as important to empower students, from enrolment throughout the programme. This 
success factor could be linked to all quality areas and domains: Management: ‘ ...you should, indeed, have a 
vision on OBL. ... but you must also make a lot more advertising for it. ... ‘yes ’ ... ‘we make informational films 
which we put on our website ’, Services: '... there are students who enroll in a programme and realise that it will 
not work (for them), in which case the counselor has a conversation with the student to redirect him/her to a 
different programme’ and products: ‘...I wrote: "create more uniformity’." Not that I like ... would like all to be 
the same will but... that students can still find their way in that course... '. 

Accessibility, linked to delivery and student support, is mentioned in not more than half of the interviews and 
three cases. Online delivery has to be accessible at all times via mainstream technology which students are able 
to use: ‘/ choose the tools that are not so difficult for them. I would like to use smartphones, but if I notice that 
there are only one or two who have one, I will choose not to use it'. Accessibility was mentioned in not more 
than half of the interviews and three cases. 

It seems that flexibility: ‘Flexibility’ related to time. When do you study, when do you learn? But also flexibility in 
terms of pace (going slower or faster through the programme)... ’ and accessibility: ‘...because one needs 
performant internet connection to be able to play al those movies... ’, are crucial to enhance access to education 
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and make participation in education convenient for students, while transparency: ‘...we try to advise students, 
we have extensive information sessions at the start of the school year. is important to inform students of 
what is expected or possible. 

Success factors related to pedagogy (productivity, personalization, interactivity and participation) are less 
prominent in the interviews. The quality dimension ‘design’ is related to the success factors productivity: ‘... for 
validity of assessment ... language training implies for example, that there are assignments that require 
individual processing by students. A set of grammatical or lexical exercises which is not an example of a good 
assignment. ’, and personalisation: ‘... assessment is no longer a purpose in itself. ... now you are working with 
the student for the added value. For their independence and for (ed. to achieve) those competencies’. 

Statements coded for online interactivity between peers is under-reported or even absent in favor of interactivity 
with content: ' ...you, give a bit of info (ed. online) ... And they must apply that. And then click on the button 
submit and they see 'what you've done now is right or wrong. ’ and teachers: ‘ ...when I give feedback I keep in 
mind that ... not in the style of "this is not good’’ or “this was a bit too weak." But that really is about what is 
wrong and what they can do to remedy it...’. 

The codes for the success factor ‘participation’ were scarce. What is mentioned is similar but not equal to codes 
related to ‘interactivity’. It seems that participation can be seen as a central success factor. It can be argued that 
decisions taken at the level of the other success factors have consequences on the way students participate in the 
educational provision: '... the way of looking 100%, because ultimately you look at some: is there participation, 
is there material available and is it being worked with, is there feedback to the students? ’. 

The theme ‘evolution’ could not be linked to a success factor. In this theme respondents state that the way OBL 
is designed and the amount of OBL in the provision evolved over time: ‘ Indeed, I think that our distance 
education and the way we use it to work has evolved tremendously ’, ‘And that really is also a choice that we 
made as an institution. And we really want to go for it. In the past it was blended learning. But now is what we 
call open CVO, in which almost the entire course is given in distance education ’. Although mentioned in only 
half of the interviews, codes for the theme ‘evolution’ were present in four cases. 

None of the themes ‘management’, ‘internal QA’ and ‘external QA’ could be linked to the success factors. What 
is mentioned focuses on the management principle of integration of the vision into the organization: '... Yes, 
distance learning and contact education should be structurally aligned to each other. That's in terms of 
documents etc... So, that the rules, are uniform, ECTS sheets are uniform... ’ Researcher: ‘And you mean are 
“aligned with each other”, it is about the provision. Not the course but,... ’, respondent: ‘Yes, both with respect 
to the courses as anything outside of the courses. Wliere do I have to go for a document when I need one, when 
do I have to do this or that? if I have to. Oh, boy sometimes at times this is very difficult... ’. Management is also 
about providing clear roles for educators, staff support and internal QA processes. In that respect respondents 
mention: ‘We therefore work with projects or project groups, ... to actually learn from collaboration...’,'... 
there is so much expertise yet every’one is still ... working on an island, I think. I think there could be much more 
cooperation ’,'...collaboration divides the work and strengthens it ... I think is a creative way of using resources. 
But yeah, that's easy for meto say, because we were forced at some time because we had to survive. But I am a 
believer of supporting micro design teams rather than individual teachers ’. 

The theme of ‘external QA’ is related to the topics of external quality assurance, verification and funding. 
Respondents report that how Rinding and external quality assurance are conducted impinges on how they 
organise OBL. Respondents complain about external QA and verification: '... now we are funded based on 
attendance. For distance education this is based on participation which is operationalised as how long someone 
is logged into the system, what they (students) have actually done is not taken into consideration, this tells 
nothing ’. 

The theme ‘integration’ is related to ‘design’. Respondents refered to different thingsabout this theme:program - 
‘distance education and face to face education should be structurally aligned ’, course/learning activities and 
assessment - ‘ The goals or better skills, which are important, how they relate to the course material. And how it 
relates to your assessment. That should actually be all in one, all in the same line, which is not so evident. ’, ‘ I've 
added here: fraud resistance. Sooner or later we're going to be caught on. How can you prove that an 
assignment is really made by student X? Oh yes, by the end through an oral exam, a jury or whatever ....’, ‘(they) 
can have their tasks made by someone else’. It seems thus that the emerging theme ‘integration’ cannot be seen 
as a success factor in its own right because it appears related to different things i.e. design and assessment. 
Researchers agreed that this is not specific for OBL. Integration also refers to how face-to-face and online 
education is structurally aligned to one another and to ‘assessment’ i.e. validity. The researchers agreed that 
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these concepts were covered by other success factors, respectively ‘productivity’ and ‘flexibility’.From the 
analysis an adapted framework is proposed, an overview of the success factors, their connections to quality 
dimensions and the number of indicators can be found in table 8. 


Table 8: list of seven success factors with tentative definitions, citations from interviews, connections to quality 
areas/dimensions and number of indicators (Appendix 1). (Kear et al., 2016; McLoughlin & Lee, 2008; Ubachs 

et al., 2007; Williams et al., 2012). 


Success Factors and Tentative Definition 


Connections to quality 
areas/dimensions 

Credibility- Credibility implies the translaties of a clean view on OBL into 
measurable targets. It implies efficient use of potential means and personnel. 

Finally, it entails the integration of these targets into the quality assurance system, 
monitoring the development of OBL and adjusting it if necessary. 

• 

• 

Management (N=13) 

Support for teachers and 
staff (N=10) 

Flexibility - The degree in which students have the possibility to fine tune 
educational needs to professional or private needs and obligations. 

• 

• 

Programme (N=4) 

Learning activity (N=l) 

"Flexibility related to time. When do you study, when do you learn? But also 
flexibility in terms of pace (going slower or faster through the programme). 
Intensity, time, intensity, learning style... ”. - A B 1 



Transparency - All initiatives taken to inform potential students about the 
programme from enrolment until graduation. 

"... also we try to advise students, we have extensive information sessions at the 
start of the school year. ...” — E L 3 

• 

• 

• 

• 

Management (N=l), 
Programme (N=4) 

Course (N=4) 

Student support (N=7) 

Accessibilty - Is determined by the online accessibilty of students and by what is 
available for them on the campus. 

• 

• 

Delivery (N=5) 

Student support (N=2) 


"that student has no internet connection ... our open learning center is also 
accessible to students. ... We ... train students in ICT skills. ... the basics like an 
on/off button of a computer. ” - C_B_1 


Interactivity - Refers to the online interaction that is supportive for the learning • Learning activity (N=6) 

process between students and the material and students and teachers. Interactivity is • Student support (N=4) 

related to design and student support. 

".../think that if you design the learning path differently it is possible to do it online. 

But the learning path is like, well like mine that I now have developed for instance 
that you don't need to do that. Wit ere you just, you're giving a piece of info and ok 
now let's apply that. And they (students) must do that. And they click on a button 
'Submit' and they see: 'Whatyou've done now is correct or is incorrect '- C_L_? 

Personalisation - The extent to which students have, and (can) make use of the • Learning activity (N=8) 

possibility to personalise (customise/maximise) their learning experience to personal • Student support (N=2) 

needs by their own choice. Personalisation ranges from personal learning (a lot of 
freedom of choice for students) to personal instruction (absence of choice). 

"One part (online) is rehearsal of exercises and implementation of what we worked 
on in class. And the second part it entails new subjects. So if for example, they have 
understood well what we dealt with during class, for all I care they can skip the first 
part that or spend less time on it. They do what they want with it. They are free to 
decide for themselves. - D_L2_2 

Productivity - The extent to which learning activities (content and assessment) are • Learning activity (N=3) 
designed to challenge/invite students in the process of knowledge creation rather 
than mere reproduction. Productivity is linked to design. 

"Yes, yes because we then surely knew: 'look, let those people (students) tell what 
they have learned and then you can dig much deeper, and really see if those 
competencies are acquired." E_B_1 

Participation - Participation is understood as the students’ active involvement in 
their learning processes. Participation is linked to Student support and Design. 

"... the way of looking 100%, because ultimately you look at some: is there 
participation, is there material available and is that being worked with, is there 
feedback to the students?" - E_B_3 
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DISCUSSION 

This study has identified a framework with success factors and indicators for quality of OBL based on interviews 
in the field of AE. The results indicate that all success factors for quality in OBL are present in AE. The success 
factors and indicators are connected to quality dimensions present in existing quality frames. While some 
success factors and indicators enable participation others have an impact on the learning process of students. We 
will discuss these findings in relation to literature on quality from the adult student perspective in HE. 

The discussion is structured in a similar manner as the findings were presented. Firstly, we will success factors 
that enable participation to education. Then we will discuss the pedagogical success factors and how they are 
related to the enabling success factors. Finally, we will address the emerging success factor, credibility. 

While flexibility and accessibility are important to increase the accessibility of adult education and facilitate 
participation, transparency is important to inform students about the posssibilitiets of the modalities of OBL. 
These findings are in line with literature. Although not much specific reference to attributes of flexibility are 
reported in AE literature, with respect to transparency emphasis is given to the importance of transparent 
communication about flexibility (Harroff, 2002; Jung, 2011). Accessibility relates to the technical requirements 
of delivery and technical support to students in order to be able to participate (Harroff, 2002; Jung, 2011; Korres 
et al., 2009; MacDonald & Thompson, 2005; Volungeviciene et al., 2014). This is in line with Grahams’ concept 
of enabling blend (Bonk & Graham, 2012; Graham, 2005; Graham & Robison, 2007) which aim to increase 
access and convenience to students.The finding that accessibility was not mentiond a lot could be explained by 
the fact that it is such a logical condition that it is easily overlooked (Ehlers, 2004). 

Success factors related to pedagogy (participation, personalisation and productivity) were less prominent in the 
interviews. While ‘personalisation’ (Dzakiria, 2012; Harroff, 2002; Jung, 2011; MacDonald & Thompson, 2005; 
Stodel et al., 2006; Zhang & Cheng, 2012) is emphasized in literature, ‘productivity’ is to a lesser extent (Jung, 
2011; Stodel et al., 2006; Volungeviciene et al., 2014). In contrast with our findings ‘interactivity’ of students is 
mentioned in relation to content, peers and faculty in literature (Dzakiria, 2012; Harroff, 2002; Jung, 2011; 
MacDonald & Thompson, 2005; Stodel et al., 2006; Volungeviciene et al., 2014; Zhang & Cheng, 2012). 
Although ‘participation’ and ‘interactivity’ are seen as distinct success factors (McLoughlin & Lee, 2008; 
Ossiannilsson & Landgren, 2012), analysis of interviews indicates that they are similar. 

The theme ‘evolution’ indicates that the centers initially focused more on success factors that relate to lowering 
the threshold to education at the expense of pedagogical success factors, but, with time came to realise the 
importance of pedagogical success factors. It suggests that pedagogical success factors: personalization, 
interactivity and productivity became more important over time for a design of OBL which invites students to 
take ownership i.e. actively participate in the learning process.This provides evidence for of Ossiannilsson and 
Landgrens’ (2012) statement that OBL is evolving towards paradigms of collaboration and networking could be 
true in AE. Findings suggest that an evolution occurred from a cognitive-behaviourist learning design towards a 
social constructivist design i.e. knowledge production and interaction (with material and teachers). It also 
suggests that institutions are challenged to take pedagogical success factors into consideration when they attempt 
to move from awareness or exploration of OBL and away from adoption and early implementation towards more 
mature implementation and growth or improvement (Graham, Woodfield & Harrison, 2013). Caution should be 
used over whether evolution in design should be interpreted as one learning theory being better than another, 
adding more to quality. It is important in this respect to take note that several generations of distance education 
pedagogy emerged over time: cognitive-behaviourist, social constructivist and connectivist pedagogy and that all 
add to quality (Anderson & Dron, 2011). 

What respondents mention in the themes ‘management’, ‘internal QA’ and ‘external QA’ corresponds with what 
Jung(2011) reports as ‘credibility’. What is mentioned is in line with what is reported in literature (Harroff, 
2002; Korres et al., 2009; MacDonald & Thompson, 2005; Stodel et al., 2006; Volungeviciene et al., 2014; 
Zhang & Cheng, 2012). It focuses on the management principles of integration of the vision of OBL into the 
organization and also about providing clear roles for educators, staff support and internal QA processes. Not 
different from a standard education programme, but with a specific focus for OBL. 

It appears that institutions that want to become successful adopters of OBL need to decide how to use their 
resources in such a way that the participation of the students is maximized. The predominant question for 
institutional QA&I process is therefore indeed if institutional, faculty and student goals are balanced (Moskal et 
al., 2013). 
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Findings indicate that institutional alignment (Moskal et al., 2013) might not be sufficient. The macro and meso 
level should be aligned too. Institutional alignment (Moskal et ah, 2013) can be hampered by how legislation 
regulates external quality assurance, verification and funding of educational institutions. Findings indicate that 
decisions taken at the macro level influence both the meso- and the micro level in institutions. The legislative 
framework provided by the government, the way external accreditation bodies assess quality, and operationalise 
attendance negatively affects funding and in this way interferes with institutional practices related to the design 
and implementation of OBL. This is in line with Ossianilsson et ah (Ossiannilsson et ah, 2015) who point out 
that “ ...Other systems (national approaches to quality) ” that “ have not considered the impact of e-learning onto 
their criteria, ” should integrate quality of OBL into their external quality frameworks to avoid “creating 
sometimes perverse results, such as limitations on the size of classrooms, or requirements for physical facilities 
which are not required for e-learning. "(Ossiannilsson et ah, 2015). Findings also stress the importance to move 
from a time-based, towards a mastery-based measurement of student performance in OBL (Graham et ah, 2013) 
for this context. 

Limitations and prospects for future research 

An evidence-based validation processes for quality frameworks is important (Inglis, 2008). The focus of this 
study lies in the development of a conceptual framework in the context of AE and relevant literature by 
determining success factors for the quality of OBL in AE and linking these to quality aspects and searching for 
indicators. 

Although all success factors for quality in OBL are present, they are distributed unevenly over the interviews and 
the frequency of mentions between success factors differs. This could be attributed to the design of the study. 
For the interviews a semi-structured interview guideline was used to give respondents the opportunity to speak 
freely with, as a possible result, the low frequency of some success factors. A rival explanation is that the 
number of interviews was not sufficient for the data to reach saturation. Yet, it can be argued that the total 
number of respondents and the number of respondents per group at policy level and programme level, should be 
sufficient to reach a point of data saturation. Other than the design of the study, it is possible that not all success 
factors are mentioned equally because an evolution is occurring in the field. Therefore, further validation by 
consultation with experts in the field is necessary. 

CONCLUSION 

A modified conceptual framework for OBL in AE has emerged from this study. Success factors with supporting 
indicators that contribute to lowering threshold to education (flexibility, accessibility, transparency, credibility) 
as well as success factors with indicators that have a direct impact on the quality of the learning process 
(interactivity, personalisation, productivity) determine participation (access) of students in education or the 
active participation of students in the educational process. The success factors and indicators are linked to 
quality dimensions and areas, present in most quality frameworks. 

This framework will enable institutions to reflect about how technologyreduce the barriersfor participation and 
how it can support active participation of studentsin theeducation process. The link of the success factors with 
the concepts of enabling and transforming blend(Bonk & Graham, 2012; Graham, 2005; Graham & Robison, 
2007)supports institutions to strategically integrate OBL in into its mission and assess the maturity of their OBL 
provision. The framework allows for growth from an enabling blend towards a transforming blend and can be 
used to determine if the provision of OBL is aligned with the needs of the students. 

Until now quality frameworks for OBL are used in addition to general quality frames in education. Yet recently 
it has been argued that OBL quality should be mainstreamed into traditional quality frameworks used by either 
educational institutions or accreditation bodies and not assessed separately (Grifoll et al., 2010; Hansson, 2008; 
Ossiannilsson et al., 2015). The framework that resulted from this study can be used to mainstream quality of 
OBL in traditional QA&I frameworks. This is possible because success factors and indicators are linked to 
quality dimensions and areas, present in most quality frameworks. 
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Appendix 1 - List of indicators 


Success factor 

Link to quality dimensions and CIPO 

Indicator 

Description of indicator 

AE Literature 
Reference* 

AE Interview 
Reference** 

1. Flexibility 

F_PR 

Quality Dimension: Design - Programme 

CIPO: Education-Curriculum-Organisation of the education 

F_1 

Duration of the programme matches the 
needs of the students 


E L 3;A L 1; 

E B 1 

F_2 

The ratio contact education vs. Online 
education of the programme matches the 
needs of the students 


D B 5;D B 2; 

E B 1;B B 1; 

C B 1;C B 2; 

C B 3 

F_3 

Students are given pacing opportunities in 
the curriculum 

LitVol; 

Lit Kor 


F_4 

Students are able to follow a flexible path 
throughout the curriculum 

Lit Sto; 

Lit Vol 


FLA 

Quality Dimension: Design - Learning activity 

CIPO: Education-Curriculum-Content of the education 

F_5 

Students are given flexibility in deadlines 
for assignments 


E_B_2; D_L2_2 

2. Accessibility 

ADEL 

Quality Dimension: Delivery 

CIPO: Absent 

A_1 

The technical infrastructure meets current 
connectivity requirements 

Lit Vol; Lit Kor 


A_2 

Students can access the learning 
environment with mainstream hardware and 
software 

LitMac 

D L2 2 

C_B_2 

A_3 

The online learning environment supports 
the intended interaction between all 
participants. 

LitVol; 

LitKor 


A_4 

The usability of the learning environment 
takes the students’ technical skills into 

account. 

LitKor 

D_L2_2 

A_5 

The learning environment accommodates 
students with special needs 

LitKor 


A_LS 

Quality Dimension: Learner Support 

CIPO: Absent (A 6) - Pupils" guidance-career guidance (A 7) 

A_6 

Students have access to technical assistance 

Lit Mac; Lit Kor; 
Lit Har 

BBJ; CBJ 

A_7 

Students have access to resources e.g. 
library, open learning center, career 
guidance,... 

Lit Jun 

C_B_1 

3. Transparency 


(Prior to enrolment in the programme) 

T_M_V 

Quality Dimension: Management-Vision 

CIPO: General-Development of vision 

T_1 

The institutional mission and vision on OBL 
is made available to prospective/potential 
students 


D_L2_2; D_L2_1 

T_ LS 

Quality Dimension: Learner support - Administrative support 

CIPO: Pupils’ guidance-career guidance 

T_2 

Prospective students are informed of 
conditions of admission 

Lit Har; 

Lit Jun 


T_3 

Prospective students are informed about 
costs related to the OBL programme 

Lit Har; 

Lit Jun 


T_4 

Prospective students are informed about 
financial aids related to the OBL 

Lit Har; 
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programme 



T_5 

Prospective students are informed about 
software and hardware requirements 

Lit Har; 

C_B_1 

T_6 

Prospective students are informed about 
required technical skills 

Lit Har; 

C_B_1 

T_7 

Prospective students are informed about the 
duration of the programme 

Lit Har; 


T_8 

Prospective students are informed about 
possibilities of personalized pathways 
through the programme 


B_B_2 


After enrolment in the programme 

H 

1 

*0 

JO 

Quality Dimension: Design - Programme 

CIPO: Education-Curriculum-Organisation of the education 

T_9 

Students experience a uniform online design 
throughout the programme/curriculum 


D_L2_2 

T_10 

The importance of online interaction 
(materials, peers, tutors) as part of the 
learning process is made explicitly clear to 
the students. 

Lit Dza; 

LitSto 

E B 2; E B 3; 

D L2 2; 

D_L2_1 

T_ll 

The intended learning outcomes are 
transparently translated in learning activities 
and assessments 

Lit Har; 

D_L1_2 

T_12 

Students are informed about tutors’ 
response time on assignments, questions,... 

Lit Har; 

Lit Dza 

B_B_1 

T_ LA 

Quality Dimension: Design - course 

CIPO: Education-Curriculum-Organisation of the education 

T_13 

Students are fully informed about the course 
requirements. 

Lit Har; 


T_14 

Students are provided with clear 
information about course assignments 

Lit Har; 

D_L2_2; A_L_3 

T_15 

Students are provided with a clear online 
course lay-out 


D_L2_1; D_L2_2 

T_16 

The lay-out of the online course is mad 
explicit to the students 


C_B_2 

4. Credibility 

C_ M_CR 

Quality Dimension: Management - Compliance and recognition 

CIPO: Context-Legislative framework-specific legislation (C 1) - Absent (C 2) 

C_1 

The educational provision meets the quality 
requirements assessed by external assessors 
(inspectorate, accreditation bodies,...) 

Lit Jun 


C_2 

The educational provision meets the 
requirements of international copyright 
legislation 


A_B_2 

C_M_L 

Quality Dimension: Management - Leadership 

CIPO: General-Leadership 

C_3 

The institutions’ management develops 
institutional standards for the design of 
OBL 

Lit Zha 

D_L2_1; D_L2_2 

C_4 

The institutions’ management develops and 
deploys the OBL provision project based 


C_B 1; D B 5 

C_5 

The institutions’ management monitors the 
implementation of the institutional mission 
and vision of OBL 


D_L2_1; D_L2_2 

C_6 

The institutions’ management gives credit to 
teachers and staff who develop OBL 

LitHar 

B_B_3 

C_7 

The institutions’ management integrates 
OBL in all aspects of the organisation 
(teachers, support staff, administration,...) 

LitMac 

A_L_2 


Management - Adequate use of resources 
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c_ m_r_l 

Quality Dimension: Logistics - finance and equipment 

CIPO: Logistics-Material management-financial means & equii 

pment 

C_8 

The institutions’ management provides 
sufficient personnel and other resources 
(infrastructure) to develop OBL adequately. 

Lit Har; 

LitMac 

C_B_2; D_B_2 

C_9 

The institutions’ management provides 
adequate resources (people and 

infrastructure) for the coaching and 
guidance of students in the OBL provision 

Lit Har; 

A_B_2 

C_10 

The institutions’ management is able to 
maintain continuity in the use of its 
applications/ ICT-tools 


C_L_4; C_L_1 

C_ M_STS_PO 

Quality Dimension: Support for Teachers and Staff 

CIPO: Staff-HRM-staff organization 


Personnel organization 

C_ll 

Job descriptions for teachers and staff 
describe their role(s) related to the 
development and optimisation of OBL 

LitVol 


C_12 

Job descriptions for teachers and staff 
include pedagogical role(s) concerning the 
OBL-coaching of their students 

LitSto 



Teams 



C_13 

The design of OBL is the result of a 
multidisciplinary team 


D_B_4; E_B_1 

C_14 

The institutions’ management fosters OBL 
learning communities for teachers and staff 

LitSto; Lit Vol 

D L2 1; D B 1; 

D B 2 


Technologv 



C_15 

Teachers and staff have access to technical 
support on demand 

LitHar 

D_B_2; D_L1_2 


Pedagogv 



Cl 6 

Teachers and staff are able to get expert 
support in OBL-pedagogy 

Lit Jun 

D_B_2 

C_ MSTSPR 

Quality Dimension: Professionalization 

CIPO: Staff-HRM-competence development 

C_17 

Teachers and staff are trained in online 
design and didactics of online learning 
activities 

LitHar 

B_B_1 

C_18 

Teachers and staff are trained in online 
communication skills 

LitHar 


C_19 

Teachers and staff are trained in 
international copyright legislation 

LitHar 

A_B_2 

C_20 

Teachers and staff have access to supply- 
and-demand driven professionalization. 

Lit Jun 


CMCQI 

Quality Dimension: Management - CQI 

CIPO: General- Quality Assurance 

C_21 

The institutions’ management has installed a 
quality assessment process that fosters a 
culture for quality improvement (e.g. 
internal and external audits) 

Lit Jun 

A B 2; B B 1; 

B_B_2 

C_22 

The institutions’ management has installed a 
quality survey process that monitors critical 
quality indicators for OBL (output measures 
e.g. drop-out, learning effectiveness, ...) 

Lit Jun; Lit Mac; 
Lit Zha 

A B 1; B B 1; 

B_B_2 

C_23 

The institutions’ management has installed a 
quality assessment process (surveys, focus 
groups, ...) in which different stakeholders 
(students, teachers, alumni, work field,...) 
are consulted 

Lit Vol; Lit Kor 


5. Interactivity 
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I_D_LA 

Quality Dimension: Design-Learning activity-with material 

CIPO: absent 

I_1 

Online Learning activities contain a variety 
of learning resources 

LitVol 

C_B_2 

I_2 

Assignments coerce students to engage 
actively with the online materials 

Lit Vol; Lit Zha 

C_L_X 

I_3 

Assignments challenge students to approach 
learning content from different perspectives 

LitVol 


I_D_LA 

Quality Dimension: Design-Learning activity-with peers 

CIPO: absent 

I_4 

Intended online interaction amongst peers 
fosters critical thinking 

Lit Har; Lit Dza 


I_5 

Intended online interaction amongst peers 
supports knowledge building 

Lit Mac; Lit Jun 


I_6 

Intended online interaction amongst peers 
fosters community building 

Lit Sto; Lit Dza; 
Lit Mac: Lit Zha 


I_D_LS 

Quality Dimension: Learner Support - with materials 

CIPO: Education-Evaluation-evaluation practice (I 7) 

I_7 

Learning activities contain a variety of self- 
assessment opportunities 


C_L_X 

I_D_LS 

Quality Dimension: Learner Support - with teachers 

CIPO: Education-Evaluation-evaluation practice (I 9)-Absent (I 8,1 10) 

I_8 

Students are supported in their ability to 
communicate online 

LitSto 


I_9 

Students are supported in their learning 
process with quality feedback by teachers 

Lit Mac; Lit Dza 

A_L_2 

110 

Online interaction between students and 
teachers fosters community building 

Lit Zha 


6. Personalization 

PeDLA 

Quality Dimension: DESIGN - Learning activities 

CIPO: absent (Pe_l,Pe_2,Pe_3,Pe_4), Education-Pupils’ guidance-guidance of the 
learning capabilities (Pe_5,Pe_6), Education-Curriculum-Content of the education 
(Pe 7), Education-Evaluation-Evaluation practice (Pe 8) 

Pel 

Authenticity of learning activities matches 
the needs of the target group 

Lit Jun 


Pe_2 

Learning activities are meaningful for the 
target group 

Lit Jun; Lit Zha 


Pe_3 

Learning activities accommodate 

differences within the target group (e.g. 
cultural differences, gender,...) 

LitHar 


Pe_4 

Students are given the opportunity to 
customize the learning activities 

Lit Sto; Lit Mac; 
Lit Jun; Lit Har 


Pe_5 

Design of learning activities allows random 
order usage 


D_L2_2 

Pe_6 

The design of learning activities contains 
remedial and deeper-level learning material 
to match students’ needs 


A_B_1; A_B_2 

Pe_7 

the design of learning activities 

accommodates the students’ different 

learning styles . 


A_B_2; A_L_2 

Pe_8 

Assessment modalities allow for active 
involvement of students 


E_B_2 

PeDLS 

Quality Dimension: DESIGN - Learner support 

CIPO: Education-Pupils’ guidance-guidance of learning capabilities (Pe_9), 
Education-pupils’ guidance-social and emotional guidance (Pe 10) 

Pe_9 

Students have access to need based learning 
support (through online tools, open learning 
centre, extra curriculum courses, ...) 

Lit Jun; Lit Har; 
Lit Dza 


Pe 10 

Students have access to psychological and 

Lit Jun 
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social support 



7. Productivity 

PrDLA 

Quality Dimension: Design - learning activity 

CIPO: absent (Pr-1, Pr 2), Education-Evaluation-evaluation practice (Pr 3) 

Pr 1 

Learning activities are problem-based 

Lit Jun; Lit Vol 

E B 3 

Pr_2 

Learners are encouraged to take an active 
role in co-constructing knowledge 

Lit Sto; LitVol 

C_L_X 

Pr_3 

Students are assessed in ways that exceed 
the mere level of knowledge reproduction. 

LitVol 

B_B_1 
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